Annotating Large Email Datasets for Named Entity Recognition with Mechanical Turk

Amazon's Mechanical Turk service has been successfully applied to many natural language processing tasks. However, the task of named entity recognition presents unique challenges. In a large annotation task involving over 20,000 emails, we demonstrate that a compet­ itive bonus system and inter­annotator agree­ ment can be used to improve the quality of named entity annotations from Mechanical Turk. We also build several statistical named entity recognition models trained with these annotations, which compare favorably to sim­ ilar models trained on expert annotations.